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Topological semimetal in a fermionic optical lattice 
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Optical lattices play a versatile role in ad- 
vancing our understanding of correlated quan- 
tum matter. The recent implementation of or- 
bital degrees of freedom in chequerboard [l|, 0] 
and hexagonal [3] optical lattices opens up a new 
thrust towards discovering novel quantum states 
of matter, which have no prior analogs in solid 
state electronic materials. Here, we demonstrate 
that an exotic topological semimetal emerges as 
a parity-protected gapless state in the orbital 
bands of a two-dimensional fermionic optical lat- 
tice. The new quantum state is characterized 
by a parabolic band-degeneracy point with Berry 
flux 2ir, in sharp contrast to the ir flux of Dirac 
points as in graphene. We prove that the ap- 
pearance of this topological liquid is universal 
for all lattices with D4 point group symmetry as 
long as orbitals with opposite parities hybridize 
strongly with each other and the band degener- 
acy is protected by odd parity. Turning on inter- 
particle repulsive interactions, the system under- 
goes a phase transition to a topological insulator 
whose experimental signature includes chiral gap- 
less domain-wall modes, reminiscent of quantum 
Hall edge states. 

The search for topological states of matter has been 
a focus of theoretical and experimental studies, since 
the discovery of the quantum Hall effect (See the review 
of Ref. Q and references therein). This problem was 
brought to the forefront again recently by the theoret- 
ical prediction and experimental discovery of the time- 
reversal invariant Z2 topological insulators in semicon- 
ductors with strong spin-orbit couplings [H-fllj]. (For 
more details see the recent reviews of Refs. [HI, [l3[ 
and references therein). For nonintcracting particles, the 
topological properties of insulators as well as topologi- 
cal superconductors have recently been classified based 
on the anti- unitary symmetries of the systems [3, EH . 
However, this elegant topological classification does not 
apply to Fermi liquid (metal or semimetal) states due 
to the existence of fermionic low-energy modes in gap- 
less systems. In this paper, we shall show, however, that 
a novel type of topologically-nontrivial semimetal unex- 



pectedly arises as a universality class for arbitrary two- 
dimensional lattices with D4 point group symmetry due 
to the mixing of orbitals of opposite parity. We believe 
that our discovery should be realizable in fermionic cold 
atom optical lattices rather easily. 
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FIG. 1: (a) The optical lattice potential in equation {TJ. 
Here we choose Vi = 2AEr and V2 = l.QEa, where Er = 
h 2 /(2mA 2 ) = h 2 /(4ma 2 ) is the recoil energy with h being the 
Planck constant, m being the mass of the particle, A being the 
wavelength of the light beam and a being the lattice constant. 
The plane at the bottom shows the contour plot of the same 
potential. The red square marks an unit cell and the green 
dots indicate the two energy-minimum points of this unit cell 
located at the bond centers, (b) The experimental setup to 
realize the lattice potential in equation (0 for V2/V1 > 1/2. 
The linear polarization of the incident monochromatic light 
beam (solid blue line) encloses an angle a with respect to 
the normal direction to the drawing plane. The black bars 
represent mirrors and the dashed arrows mark the x and y 
directions of the coordinates. See Methods for analysis. 
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The physics of higher orbtials in optical lattices has 
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FIG. 2: The single-particle energy spectrum (measured in 
the unit of En) for the lowest four bands and the topologi- 
cal structure near band degeneracy points, (a) and (b) show 
the band structure for the momenta along the contour from 
F to M to X and back to F. This contour is shown in the 
inset in (a) represented by the blue lines, while the red square 
marks the Brillouin zone. At V\ = 2AEr, two different types 
of band structures are observed, (a) shows the band struc- 
ture at Vi < 0.87Er (V2 = OAEr), where the hybridization 
between different orbitals is weak. We refer to this type of 
band structure as the weak-hybridization limit. The band 
structure at larger V2 was shown in (b) with V2 = 1.6Er. 
This case is referred to as the strong- hybridization limit. The 
dashed line in (b) marks the chemical potential, at which the 
system becomes a topological semimetal. The marginal case 
V2 = is shown in the Supplementary Information, 

where all the three upper bands touch at T point. 



recently emerged as an exciting new front in both theo- 
retical and experimental (e.g., early [l9l - l22l | and 
recent [l|-y|) studies. We specifically examine a model 
system that resembles the D 4 symmetric double- well lat- 
tice reported by the Hamburg experimental group p], [|J , 
but our conclusions apply generally to other lattices with 
the same point group symmetry. Consider the optical 
lattice shown in Fig. QJ, with the potential 

V(x,y) = —Vi[cos(kx) + cos(ky)] 

+V~2[cos(kx + ky) + cos(kx — ky)]. (1) 

Here, k — 2-n/a and a is the lattice constant, x and 
y are the 2D coordinates in configuration space. The 
parameters V\ and V% are chosen to be positive. This op- 
tical lattice can be formed using a single chromatic light 
field following the experimental setup shown in Fig. [TJd 
for V2/V1 > 1/2. For completeness, below we will first 
consider the general situation with V2/V1 > 0. Then, 
we will show that the parameter range of interest in our 
work is V2/V1 ~ 2/3 > 1/2, which can be realized using 
the proposed experimental setup shown in Fig. lb and 
discussed below in the Methods section. 

For V2 = 0, the V\ term induces a square lattice with 
lattice constant a. As V2 increases, the potential energy 
at the center of a unit cell [with coordinates (0,0)] is in- 
creased while the potentials near the bond centers [with 
coordinates (±a/2,0) and (0,±a/2)] are reduced. For 
V2 > Vi/2, each unit cell contains potential minima lo- 
cated at (±a/2, 0) and (0, ±a/2) as shown in Fig. [Tk, 

We numerically solve the band structure of this lattice 
via plan-wave expansions and find that band degeneracy 



points appear between higher orbital bands at T and M 
points (the center and corner of the Brillouin zone). For 
the lowest four bands, as shown in Fig.rjl in the small V2 
limit, the second and third bands cross at both V and M 
points. For larger V2, there are still two band degeneracy 
points for the lowest four bands, but now the second and 
third bands only cross at M, while the third and fourth 
bands become degenerate at T. For even larger V2 (not 
shown), the first and second bands become degenerate at 
M, while the third and fourth bands touch at T. This 
large V2 limit is dominated by the same physics as in the 
intermediate V2 regime, and thus, we will only focus on 
the small and intermediate V2 in this paper. 

The band degeneracy phenomenon described above is 
generic and stable. In fact, as shown in the Supplemen- 
tary Information, for noninteracting particles, these band 
degeneracy points are topologically protected and remain 
stable when system parameters are tuned adiabatically, 
as long as the lattice point group symmetry is maintained 
(although a band degeneracy point may move from be- 
tween the n and n + 1 bands to the m and m 4- 1 bands as 
shown in the examples above) . With details presented in 
Methods and the Supplementary Information, near the 
band degeneracy point, a 2D vector field (ht) in momen- 
tum space can be defined using the Hamiltonian of the 
system. At the momentum fc, the length of this 2D vec- 
tor (\hj^\) gives (half of) the energy splitting between two 
energy bands. For the band degeneracy points in our 
model, this vector field possesses a topological defect, 
a vortex with winding number 2. At the vortex core, 
the length of the vector vanishes (|/i^| = 0), indicating 
that the band gap vanishes here (i.e., a band degeneracy 
point). It is this topological property that dictates the 
stability of the band degeneracy against any adiabatic 
deformation. From the mathematical point of view, this 
nontrivial topology can be described rigorously using the 
topological index of the Berry flux, which is 2-7T for this 
case. 

In addition, the band degeneracy point is also pro- 
tected by the parity of the Bloch wavefunctions under 
space inversion. In fact, as shown in the Methods section, 
it turns out that all the essential physics of the topolog- 
ical semimetal can be understood within a simple tight- 
binding picture without considering the full band struc- 
ture theory, and the key ingredient for this phenomenon 
is the mixing between the orbitals of opposite parity. In 
the particular model we consider here, the semimetal is 
formed by the hybridization between the d orbital and 
the two p (p x and p y ) orbitals at each lattice site. 

We now study the instability of the topological 
semimetal in the presence of interaction, with details pre- 
sented in Supplementary Information. We start with the 
tight-binding Hamiltonian and derive an effective low- 
energy theory around the Fermi point, which in this case 
is the degeneracy point of the third and fourth band 
(Fig. rjja) • It turns out that this effective theory in the 
presence of interaction can be mapped onto a general the- 
oretical model of d-w&ve symmetry which was analyzed 
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in Refs. [23|, [24| via the renormalization group technique. 
Therefore, by mapping the results back from that d-wave 
model, we obtain the universal property for the band de- 
generacy point of the topological semimetal we present 
here. Below, we summarize the main results. 

As temperature is lowered below a critical value, T c , 
the system undergoes a second order phase transition, 
where T c ~ We~ a/NWV with N(0) the density of states 
at the chemical potential, V the interaction strength and 
W the band width. The parameter a is a dimensionless 
constant whose value is determined by the band struc- 
ture. In our model, the order parameter describing this 
low-temperature ordered phase is the z-component of the 
angular momentum (Lp) = -i(j? x? p v ,p- p y? p x ,r) , where 
p Xt f and Py t p are the fermion annihilation operators of 
the p x and p y orbitals on site r. (This order parameter 
can be mapped to the order parameter $ in the general 
theory studied in Ref (23j.) 

In our system, the repulsive interaction can be refor- 
mulated as 



flint = VY;Pl,?Px,rPlfPy,r- 



(2) 



where V > is the interaction strength. This interac- 
tion term favors a state with nonzero angular-momentum 
(L z ) ^ 0. In an ordinary metal or insulator (or graphene 
[25[ ) , the formation of nonzero angular momentum costs 
kinetic energy which usually dominates over the energy 
gain from interaction unless the interaction strength is 
very large. However, for the topological semimetal we 
find here, the energy cost for nonzero L z from the ki- 
netic part is always subleading compared with the energy 
gain from interaction at low enough temperature. This 
results in the spontaneous generation of angular momen- 
tum, which is a key theoretical insight of our work. 

From the symmetry point of view, this low temper- 
ature phase spontaneously breaks the D4 point group 
symmetry down to C4, and also breaks the time-reversal 
symmetry. This symmetry breaking pattern belongs to 
the Ising universality class resulting in two degenerate 
ground states with opposite angular momentum. 

As for the band structure, the band degeneracy at T 
is lifted in the symmetry broken phase. (The degener- 
acy at M is also lifted. However, this is not relevant to 
our study since that degeneracy point is located far below 
the chemical potential.) Hence the topological semimetal 
becomes a fully gapped insulator in the presence of in- 
teraction with the gap being V(L Z ). 

This insulator turns out to be topologically nontrivial 
characterized by the nontrivial value of the topological 
index, known as the first Chern number. The Chern 
number for this state is 1, which indicates that this sys- 
tem shares the same topological properties as the quan- 
tum Hall state with filling 1. However, in contrast to 
the quantum Hall effect where the nontrivial topologi- 
cal state is induced by a strong external magnetic field, 
here the same quantum topological state of matter orig- 
inates from many-body effects in the absence of any ex- 




FIG. 3: Topologically protected edge states and domain-wall 
modes. Figure (a) shows the single-particle energy spectrum 
of the insulating phase with (L z ) 7^ computed within mean- 
field approximation on a cylinder geometry (see Supplemen- 
tary Information for technical details). The horizontal axis 
is the momentum defined along the periodical direction of 
the cylinder (from —n/a to n/a) and the vertical axis is the 
energy. The pink curves at the bottom of (a) describes the 
states in the valence bands filled by particles, while the blue 
curves on the top part are the empty band. The green and 
red curves are the chiral gapless edge states located on the 
two edges of the cylinder as shown in (b). Here, figure (b) 
is a schematic picture showing the geometry of the system 
we used to compute the edge states. The black solid lines 
show the underlying square lattice. The two thick lines at the 
edges (red and green) represent the chiral edge states, with 
arrows indicating the chirality. The length of the cylinder we 
used is 30a with a being the lattice spacing. In this case, the 
finite-size effects are negligibly small. 



ternal magnetic field. In general, states with nonzero 
Chern number in the absence of an external magnetic 
field are known as the anomalous quantum effect states, 
first proposed in a toy model on a honeycomb lattice by 
Haldane (26|. Recently, several different possible realiza- 
tions of the Haldane model in cold gases were discussed 
using lattice rotations [27j or light-induced vector poten- 
tials [281 ] . In our predicted topological phase, however, 
interaction plays a decisive role in sharp contrast to the 
noninteracting situation prevailing in the quantum Hall 
effect or anomalous quantum Hall effect. To the best of 
our knowledge, our work is the only theoretical predic- 
tion in the literature of an interaction-driven anomalous 
quantum Hall state. 

Furthermore, if two spin components are both present 
in the atomic gases, the same interaction effect may 
lead to a time-reversal invariant Z2 topological insula- 
tor. This phenomenon can be partially understood as an 
interaction-driven 2D-version of HgTe. As pointed out in 
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Ref. [8j, the combined effect of spin-orbit coupling and 
strain opens a gap at a 3D quadratic band degeneracy 
point and leads to a 3D topological insulator. By con- 
trast, in our 2D system, topological states arise purely 
due to many-body interaction effects. 

To further demonstrate the topological nature of this 
insulating phase, we computed the band structure of this 
state on a cylinder, as shown in Fig. [3] Here, although the 
bulk modes are all gapped, there is a gapless topological 
chiral edge state on each of the two edges of the system, 
which is the direct signature of a topologically nontrivial 
insulator. 

The phase transition being discussed in our work has 
strong analogy to the BCS theory of superconductiv- 
ity. In particular, the two classes have similar scal- 
ing formula for the mean-field transition temperature 
(T<7 ~ e a / N (°Wy However, the phase transition here 
breaks only a discrete symmetry (time-reversal) and thus 
belongs to the Ising universality class. In 2D, the fluctu- 
ation effect is weak for an Ising transition and long-range 
order is sustained at finite temperature. On the contrary, 
the BCS transition breaks the continuous U(l) symmetry 
and belongs to the XY universality class. As a result, 
the BCS transition in 2D is a Kosterlitz-Thouless transi- 
tion, whose transition temperature is strongly suppressed 
by phase fluctuations and is much lower than the mean- 
field prediction. Thus the transition temperature for our 
problem should be much higher than the BCS transition, 
if all other parameters (N(0), V, etc.) have the same 
value. Therefore, under equivalent conditions the phase 
transition predicted by us should be much easier to ob- 
serve in 2D optical lattices than the corresponding BCS 
Kosterlitz-Thouless transition. 

Beyond its theoretical significance, the topological 
semimetal state also has robust and unique experimental 
signatures. For example, the energy band structure of 
the unique band-crossing degeneracy point can be de- 
tected directly u sing experimental techniques, such as 
Bragg scattering [291 ] as discussed in our Supplementary 
Information. 

At low temperatures, the system remains a topologi- 
cal semimetal for attractive interactions but becomes an 
insulator for repulsive interactions. Since both the val- 
ues and the signs of interaction can be tuned in ultra- 
cold gases, this phase transition, between a compressible 
liquid and an incompressible insulator, can be studied 
experimentally by measuring the compressibility at dif- 
ferent interactions. In addition, Bragg scattering can also 
be used to detect the insulating gap induced by the re- 
pulsive interactions. Because the low-temperature topo- 
logical insulating state spontaneously breaks the time- 
reversal symmetry, any experimental measurements sen- 
sitive to the time-reversal symmetry can also be used to 
identify this phase, such as the Hall effect. 

The direct experimental evidence for a topological in- 
sulator is the gapless chiral edge state which is a metallic 
state localized on the edge of a topologically-nontrivial 
insulator. However, it is worthwhile to note that the 



sharp edge in the condensed matter system is absent in 
cold atomic gases. Due to the existence of the slowly 
varying trap potential, one expects the density to de- 
crease away from the center of the trap. Therefore, the 
system is a liquid near the edge due to the low filling frac- 
tion. This liquid state from incommensurate filling will 
hybridize with the topological edge state, which makes 
the observation of the topological edge states challenging 
in atomic systems. This difficulty can be avoided if two 
domains of topological insulating phases with opposite 
angular momenta are induced. At the domain wall be- 
tween these two areas, compressible chiral domain-wall 
states should exist. Since this domain wall can be chosen 
to locate near the center of the trap, far away from the 
trivial liquid state near the edge of the system, it should 
in principle provide a clean signature for the topological 
edge states. These domain-wall modes can also be de- 
tected using Bragg scattering, where one finds that the 
insulating gap is reduced to zero near the domain-wall. 
In each real experimental system, due to finite number of 
particles on a particular optical lattice, the vanishing of 
the insulating gap at the domain wall is in fact prohibited 
by finite size effects. For topological insulators, such fi- 
nite size effects have been systematically studied and the 
metallic edge states are found to be detectable even for 
a system with about 10 particles [30]. An alternative ex- 
perimental way of seeing the topological edge state would 
be to have a sharp trap boundary, as in a square- well po- 
tential, which would suppress the hybridization between 
the trivial liquid phase and the edge topological state. In 
such a square- well trap, the topological edge state should 
manifest itself directly. 



Methods 

Creation of the optical lattice. In the experimen- 
tal setup shown in Fig. [TJd of the Letter, by superimpos- 
ing two monochromatic optical standing waves oscillating 
in phase, we implement the electric field 



E=e\ -h= 



smcn 



ana 



V2 

cos a 



cos [k(x + y)/2] 



7f smaN 
' I ^=sina 

cos a 



cos [k(x — y)/2] . 



(3) 



The corresponding light shift potential is U(x, y) = 
— x\E(x, y)\ 2 with \ denoting the real part of the polariz- 
ability. It is straight froward to check that this potential 
is identical to the potential we proposed in the letter, up 
to a trivial constant: 

U(x,y) = — Vi[cos(kx) + cos(ky)} 

+ V2[cos(fca; + ky) + cos(kx — ky)] — \e 2 , (4) 
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with 

V\ = - xe 2 cos 2 a, (5) 
V 2 =- X e 2 /2- (6) 

By choosing blue detuning, i.e., \ < 0, we obtain V± > 
and V<z > 0. When the polarization direction, a, is 
changed, the ratio V2/V1 can be tuned to any value above 
1/2. For example, using fermionic potassium K with 
a principle fluorescence line at 767nm, a standard green 
frequency-doubled Nd:YAG-laser (532 nm) would be a 
suitable light source for implementing the desired optical 
potential. 

Hybridization between orbitals of opposite par- 
ity. It turns out that all the essential physics of the 
topological semimetal can be understood within a sim- 
ple tight-binding picture and the key-ingredient for this 
phenomenon is the mixing between orbitals with oppo- 
site parities under space inversion. Here, we outline the 
main procedures and results of calculation for the mixing 
of parity even d x i_ y i and odd p x and p y orbitals, and de- 
fer the details (e.g., model Hamiltonian, band structure, 
etc.) to the Supplementary Information (Section S-6). In 
this study, these three orbital bands are considered next 
to the chemical potential level and all other orbitals are 
assumed to be far separate from them (such that their 
effects can be dynamically ignored). When the mixing 
between the two types of orbitals is weak, the parity-odd 
orbitals form two bands, which cross each other at the T 
and M points, while the band formed by the parity-even 
orbitals show no degeneracy (similar to Fig. ®l which 
was obtained by numerical diagonalization). In contrast, 
as the mixing between different types of orbitals is en- 
hanced, the three bands formed by these three orbitals 
hybridize together, and now the middle band crosses with 
both the other two bands, one at T and another at M, 
similar to Fig.[2h. In fact, the top three bands shown in 
Fig.[2]are mainly contributed by thep x , p y and d orbitals. 
In the Supplementary Information, a full comparison is 
provided between the band structure of the optical lattice 
model defined by the potential (JXJ) and that of the effec- 
tive three-orbital (p x ,Py, d x 2_ y 2) tight-binding model. 



Instability under infinitesimal repulsion. Us- 
ing the conclusions from Ref. [23| . we found that under 
renormalization group, the repulsive interaction shown in 
equation ^ is a marginally-relevant perturbation and it 
is also the only relevant perturbation for spinless fermions 
with short-range interactions. Therefore, at low temper- 
ature, this interaction term dominates the low-energy 
physics and will stabilize a state with nonzero angular 
momentum (L z ) 7^ 0. This state is a topological insula- 
tor with Chern number 1, in agreement with the general 
study shown in Ref. [23| . This conclusion is further ver- 
ified in Fig. [31 where we examined the mean-field single- 
particle spectrum for a cylindric geometry and observed 
the gapless chiral edge states. 
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Supplementary Information 

S-l. OPTICAL LATTICE 

Figure [Si] shows the optical lattice described by 

V(x, y) =— Vi[cos{kx) + cos(fcy)] + l^[cos(fca; + ky) + cos(kx — ky)}. (SI) 

In Fig. ISlb . we employed the same parameters used in the letter to derive the topological semimetal. For V2 < Vi/2 
[Fig. ISll(a)]. the optical lattice is basically a simple square lattice with a single energy minimum in each unit cell 
located at the center. However, when V2 > V\/2 [Fig. ISll (b) and (c)], there are two energy minima in a unit cell, 
located at the bond-center. As for the marginal case V2 — V\/2 (not shown), the minimum of the potential is reached 
when x or y becomes 2nn/k with n being any integer. 



S-2. PLANE- WAVE EXPANSIONS AND NUMERICAL CALCULATION OF THE BAND STRUCTURE 



We expand the Bloch wavefunction in the basis of plane waves 

\I/2(r) = a n me i ( 27Tn / a + k n:) x + i (' 27Tm / a + k y)y (g2) 

n,m 

where a n , m are the complex coefficients with n and to being integers and r = (x, y) is the real-space coordinates. In 
the lattice Hamiltonian, the potential energy introduces hybridization between the plane-waves with different n and 
to. For the low-energy bands, the contributions from plane waves with large to and n are small. Thus, we ignore 
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FIG. SI: The optical lattice shown in equation (|S1|) at different parameters. The darker (lighter) regions represent areas where 
the potential is low (high). The dashed line marks one unit cell of the lattice. In (a-c), Vi = 2AEr and V2 ~ OAEr, 1.6Er 
and 3.2Er. 
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FIG. S2: The single-particle energy spectrum of the lowest four bands. Here, we show the band structure for the momenta 
along the contour from F to M to K and back to F, as shown in the inset of Fig. (a). The parameters here are chosen to be 
Vi = 2AE R while V 2 takes the value of (a) 0AE R , (b) 0.87E R , and (c) 1.6E R , respectively. 



states with n and m larger than certain cutoff N. In the numerical calculation demonstrated here, N ~ 20. And the 
band structure is independent of the choice of N for N > 10 (up to negligible small corrections to the eigenenergy.) 

Using this approach, the Hamiltonian is a (2N + l) 2 x (27V + l) 2 matrix. We find the eigenvalue and eigenvector of 
this matrix at each momentum in the Brillouin zone. The eigenvalues (as a function of k) provides the single-particle 
band structure, while the eigenvector gives the coefficients of a„ jm in equation (|S2j). In Fig.[S2] the band structure is 
shown at V\ = 2AEr for different values of V2. 



S-3. DEGENERACY OF PARITY ODD ORBITALS IN A POTENTIAL WELL WITH FOUR-FOLD 

ROTATIONAL SYMMETRY 

Before studying the band degeneracy on a lattice, we examine the degeneracy of orbitals in a single potential well. 
The conclusions discovered here shall be generalized to the study of band degeneracy in the next section. 

Consider a particle trapped in a 2D potential well with four-fold rotational symmetry. The symmetry group 
describing this rotational symmetry is the dihedral group of D4. We define </> m (r) with m — 1,2,... to be the 
eigen-wavefunctions. Under a certain rotation in the symmetry group, R £ D4, the wavefunctions 4> m (r) reads 

ct> n (f f ) = J2 U n,m( R ) ( t> m (^ (S3) 
rn 

where U n . m (R) is the (n,m) element of the unitary matrix U(R). By definition, the matrices U(R) for all rotations 
in the point group forms a unitary representation of the symmetry group D4. 
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For any unitary representation of D4, it can always be separated into the direct sum of irreducible representations. 
In terms of the matrix U (R) , this means that it can be block diagonalized. Each block corresponds to an irreducible 
representation of D4. For a n-dimensional irreducible representation, the block has the size of n x n. 

In general, the states within a block (which belongs to the same irreducible representation of D4) are related to 
each other via rotation. Since the system is invariant under D4, these states must have the same energy. Thus, for a 
state that belongs to an irreducible representation of dimensions n > 1, it must be n-fold degenerate. 

For the D4 group, there are 4 one-dimensional representations (known as A%, A%, B\ and B2) and 1 two-dimensional 
one (E). Interestingly, all the one-dimensional representations of D4 are even under space- inversion 

TO = 4> m (-f). (S4) 

On the other hand, the two-dimensional representation E has an odd parity 

TO = -<F(-r1- (S5) 

Therefore, we conclude that any space-inversion odd state in this system is two-fold degenerate, while space-inversion 
even states are in general nondegenerate. 



S-4. SPACE-GROUP SYMMETRY AND BAND DEGENERACY POINTS 



The crystal symmetries of a lattice are described by the corresponding space group G. Each element of the space 
group g = {R\v\ is a symmetry operation which combines a space rotation R (represented by a d x d orthogonal 
matrix with d being the spatial dimensions of the system) and a space translation v (represented by a (i-dimensional 
vector). For a spatial vector r, 

gr = {R\v}r = Rf + v. (S6) 
The product of two group element is defined as 

gg' = {R\v}{R'\v'} = {RR'\v + Rv'}, (S7) 
and the identity operator of the group is 

e = {710}, (S8) 

where I is the identity matrix and is the vector with zero length. 

The elements of the space group with v = forms a subgroup of G, which is the point group of the lattice. Another 
important subgroup is formed by the elements with R = I, which gives all the space translations commensurate with 
the lattice vector. By definition, equation (|S7[) implies that the space group is the semidirect product of the point 
group and the lattice translational group. 

Consider the Bloch wavefunction ^g(r), where n is the band index and k is the momentum. Here we drop the 
spin index, which is irrelevant for our discussion. Under the transformation g = {R\v}, the Bloch wavefunction at 
momentum k transfers into the Bloch wavefunction at momentum Rk: 

m 

where U- g is a unitary matrix. 

For each momentum k, the rotations which keep k invariant (Rk — k) form a subgroup of the point group of the 
lattice. This subgroup is referred to as the little group Gg. For the little group Gg, equation (|S9|) reduces to 

R^ = J2 U R"^f^- ( S1 °) 

rn 

Here the unitary matrices U R ? form an unitary representation of the little group Gg. 

In a square lattice, the point group is D4. For the Y point [k = (0, 0)] and the M point [k = (ir/a, n/a)], the little 
group coincides with the point group D4. Therefore, at any of these two momentum points, we can repeat the same 
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procedure as discussed in Sec. IS-31 Similar conclusions are found concerning the phenomenon of the degeneracy. 
Namely, at the T or M point, if any of the Bloch waves is odd under space-inversion, 

*'i(r) = (Sll) 

the states must be two- fold degenerate (i.e. two bands must cross at this momentum point). On the contrary, if the 
Bloch waves are even in parity, 

*g(f) = *g(-f), (S12) 

they are in general nondegenerate. 



S-5. DEGENERACY AND PARITY OF ORBITALS 



As shown above, the band degeneracy at T and M are determined by the parity of the Bloch wavefunctions under 
space inversion. Here we show that this parity can be deduced from the parity of the orbitals which form the energy 
band. 

In the tight-binding picture, the orbitals at each site forms energy bands as tunnelings are introduced. Along this 
line of thinking, the Bloch wavefunction ^{r) can be written as a superposition of local orbitals as: 

R,n 

where n is the band index and m indicates the orbitals on a single lattice site. (j> m (r) is the wavefunction for orbital 
m at certain site and c„. m is a complex coefficient. ./V in the normalization factor is the total number of sites of the 
lattice and R gives the coordinate of the sites of the Bravais lattice. 
Under the space inversion, /, we have 

R -> -R, (S14) 
r->-f, (S15) 

k->-k. (S16) 

At r and M, e lk ' R — ±1 and thus is invariant. Therefore, as can be seen from equation (|S13[) . the parity of a Bloch 
wavefunction at these two points is dictated by the parity of the orbitals that form this Bloch wave. In the particular 
case studied in the main text, the band formed by space-inversion odd orbitals are doubly degenerate at T and M. 
This will be demonstrated in details using a specific tight-binding model in Sec. IS-61 



S-6. HYBRIDIZATION BETWEEN ORBITALS OF OPPOSITE PARITY 



A. Tight-binding model 



Considering a square lattice with three orbitals on each site (p x , p y and d x a_ y a). The Hamiltonian of the tight- 
binding model takes the following form 

H = - t dd ^2(didr>+s x + d\d? +Sv + h.c.) 

r 

+ t pp Y,(pl,f>+St x Px,? + Pl,r+a y Pv,r + h.C.) - t' pp Y,(pl,r+a v P^f + p\ }P+Sx P V ,f + h.C.) 
r r 

+ tpd EfrW*.** ~ Pl,r d r+S* + Pl,r+a v d r - p\^+S y + h.C.) 
r 

+ (S17) 

r 

where r is the coordinates of the lattice sites; a x (a y ) is the lattice vector in the x (y) direction, and p x ,r, Py,r and dp 
are the fermion annihilation operators for the p X7 p y and d x 2_ y 2 orbitals at site r. The hopping between orbitals on 
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FIG. S3: The single-particle energy spectrum in the tight-bind model. The band structure of (a) is the weak-hybridization 
case, while (c) shows the strong- hybridization regime. The marginal case is shown in (b). Here, we choose the hopping strength 
tdd = tpp = t p d — 1 and t' pp — 0.2. From (a) to (c), the values of S are 10, 5.6 and 1. The tight-binding model captures 
qualitatively the band structure of the second, third and fourth band shown in Fig. [S2] 



neighboring sites is described by the hopping amplitudes tdd, t pp , t p d and t' pp . The sign and strength of the hoppings 
are determined by the over-lap between orbitals. In our convention, all the hopping strengths here are positive and 
t' pp is the smallest. The last term gives the energy difference between p and d orbitals for a single site problem 6, 
which we assume to be positive. 

In the momentum space, the tight-bind Hamiltonian becomes 

k V Py.k J 

where dg, p x g and p y £ are the fermion annihilation operations at momentum k and 

( — 2tdd(cos k. x + cos k y ) + S 2it p dSmk x 2it p dSmk y \ 

-2it pd sink x 2t pp cos k x - 2t' pp cos k y . (S19) 

—2it p d sin k x 2t pp cos k y — 2t' pp cos k x J 

For simplicity, we choose the lattice constant a = 1. 

The eigenvalues of the 3x3 matrix in equation (|S19[) gives the band structure. Depending on the value of 5 
(compared with the hopping strength), two different types of band structures are found as shown in Fig. IS3I For 
5 > 4:tdd + 2t pp — 2t' pp , the weak hybridization is observed, but for 5 < Atdd + 2t pp — 2t' pp the strong- hybridization 
regime is reached. For the marginal case at S = Atdd + 2t pp — 2t' pp , the three band touch at T. This recovers the band 
structure calculated from plan wave expansion for the optical lattice presented in the main text, if we identify these 
three bands as the second, third and fourth band there. 



B. Generalization 



It turns out that the conclusion above can be generalized to other orbitals of opposite parity. To demonstrate this, 
we consider a general tight-binding model on a square lattice. We further assume that at each site, only one parity 
even orbital and a pair of parity odd orbitals are close to the chemical potential, so that we can ignore all other 
orbitals in the leading order approximation. Here we consider two parity odd orbtials because for any potential wells 
with four-fold rotational symmetry, the odd parity orbitals are doubly degenerate, while the even parity orbitals are 
in general nondegenerate. We also assume the energy difference between of the parity-even and parity-odd orbitals is 
described by a parameter S. 

When 5 is much larger than the hopping strength, the parity-even and parity-odd orbitals are separated in energy. 
Therefore, the mixing between these two types of orbitals is small. Roughly speaking, in this limit, the parity-even 
orbitals will form one band, which in general has no degeneracy point. The parity-odd orbitals form two bands, which 
cross each other at T and M, due to the odd parity of the orbitals. This is the weak strong-hybridization limit. 

On the other hand, when 8 is small compared with the hopping strength, the hybridization between parity-even 
and parity-odd becomes important. In this case, it is no longer possible to identify the distinct parity-even band and 
parity-odd bands, since all the three bands are now mixed together. Here, the upper two bands cross at T (or M) 
and the lower two bands cross at M (or T), which gives us the strong- hybridization limit. 
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S-7. TOPOLOGICAL PROTECTION OF THE BAND DEGENERACY POINTS 

In this section we provide two equivalent topological indices to prove the topological nature of the band degeneracy. 
The first approach shows that the band degeneracy point can be considered as a vortex in the momentum space with 
integer winding numbers, and the second one presents the same topological structure through the concept of the Berry 
flux 31, 321. 



A. Band degeneracy as a topological defect in momentum space 

Consider the tight-binding model presented in equation (|S18|) . Near the band degeneracy point at T, it is easy 
to check that the energy of the fermion modes from d orbitals are far away from the chemical potential, so we can 
integrate out these high-energy degrees of freedom and focus only on the low-energy modes (from p orbitals). By 
doing so, the Hamiltonian is reduced to an effective two-band model with 

* o= ?( p i.s*l.0*(£j)- (S20) 

k 

Here, although the d orbitals drop off from the Hamiltonian, the kernel of the Hamiltonian, H, receives corrections 
from virtual processes in which a particle jumps from a p orbital to a d orbital and then back to a p orbital (and other 
higher-order virtual processes, which we would not consider here). Using perturbation theory, the matrix % can be 
determined order-by-order as 

■y _ ( ^22 %23 \ _ 1 I %21%13 \ , /or,, n, 

\H 32 H33 J H u — fi\ n 3 iHi2 H 31 Hi 3 J + ■ ■ • 1 > 

where [i is the chemical potential and Hij is the component of the matrix shown in equation (|S19I) . Here, the 
first term on the right hand side is the zeroth order term in the perturbation expansion, generated by direct hoppings 
between p orbitals, while the second term is from the second order perturbation, which describes the virtual hopping 
processes mentioned above. 

By expanding the momentum around the T point, to the leading order, equation (|S20|) becomes 

H o-X,{ K,% Pyl ) { 2 t 3 k x k y tiAj + t 2 kl ) \ p y [ K J ' (S22) 
fc 

where 



U 2 pd 

h=tpp+ 2t pp -2t> pp +4t dd -6> (S23) 
ti = -t' pp , (S24) 
2t 2 

2t pp - 2V pp + 4:t dd - d 

This Hamiltonian recovers the general model of a quadratic- band crossing point studied in Ref. [23| ■ 

The 2x2 matrix in equation (|S22I) can be expanded in the basis of four independent matrices of SU(2) algebra as 
follows 

U = t -^(kl + k»)I + 2t 3 k x k y a x + t -^(kl - (S26) 

where a x and a z are the two Pauli matrices and I is the 2x2 identity matrix. We can define a 2D planar vector 
using the coefficients of the two Pauli matrices 

h = (2t 3 k x k y , t -^-^(kl - k 2 y )) . (S27) 
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FIG. S4: Band degeneracy point as a topological defect. Here (a) shows the planar vector h defined in equation (|S27|I in the 
momentum space near the band degeneracy point that we found in the square lattice model. For comparison, we show in (b) 
the same planar vector near the Dirac point as in graphene. In (c), we show that the band degeneracy point we found actually 
corresponds to a vortex in the momentum space with winding number 2. Here the vector field H is defined in equation QS270 . 
For comparison, we also show the same vector field for a Dirac point, which corresponds to a topological defect with winding 
number 1. This nontrivial winding number indicates that the band degeneracy is topologically protected and thus can not be 
lifted in adiabatic procedures, unless the symmetry of the system is broken. 



The length of this vector h has the physical meaning of (half of) the band gap between the two bands, if one notice 
that the dispersion relation for the two bands are 



t\ + *2 2 



(ki + k^)±\h\. 



(S28) 



As shown in Fig. IS41 this 2D planar vector h has a vortex structure at k = 0. A vortex is a topological defect 
described by the topological index (the winding number), 



hx^ ( hy_ 

\h\ \\h\ 




(S29) 



For the topological semimetal which we consider here, the winding number is W = 2. For comparison, the vortex 
structure for a Dirac point is also plotted, which has winding number 1. 

At the vortex core, h = 0, which indicates a band degeneracy point. Thus, the band degeneracy can be considered 
as a topological defect in the momentum space. It is this topological nature that protects the band degeneracy that 
we have found from being lifted against any adiabatic deformation of the Hamiltonian. 

In fact, this vortex structure is generic for symmetry protected band-degeneracy point. In general, if we focus on 
the two bands of the degeneracy point and treat the system as a two-band model, the Hamiltonian can always be 
expanded into the identity matrix and the Pauli matrices, as in equation (|S26|) . In the presence of the time- reversal 
and space inversion symmetries, at most two Pauli matrices can appear while the third one is forbidden due to 
symmetry. Thus, we can use the coefficients of the two Pauli matrices to form a 2D planar vector as in equation (jS27[) 
and the length of this vector gives the band gap between the two bands. 



B. Berry flux and degeneracy points 

The same topological nature can be presented in terms of the Berry flux. The Berry flux is defined as the contour 
integral of the Berry connection in the momentum space (3lL |32| . 

$g = j> dk-A 7 i, (S30) 
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where n is the band index and C is certain closed contour in the momentum space. Here, the Berry connection A 7 ^ is 
defined as 



Al = -ij d^i(ry V^2(r), (S31) 

with ^g(r) being the Bloch wave of band n at momentum k. The Berry connection A 7 ? can be roughly considered 
as a gauge field in the momentum space. In this analogy, the Berry flux corresponds to the magnetic flux in the 
area enclosed by contour C. Similar as the flux quantization of the magnetic field, the Berry flux of a time- reversal 
invariant system is quantized to rrnr with m being an integer |3lL l32j | . 

However, in the definition of the Bloch wavefunctions, there is an undetermined U(l) phase that can be chosen 
arbitrarily. This can be seen by noticing that for any Bloch wave ^(r), the following equation, 

|r™(r) = e ^ ( %]J(r), (S32) 

also defines a Bloch wave where 4>{k) is an arbitrary function of momentum k. For any <fi(k), the integral over a closed 
contour C in the BZ, 

W = j> dkV z <j)(k), (S33) 

always leads to an integer value due to the periodical structure around the contour. 

For a general contour C, the redefinition of phase in equation (|S32[) changes the value of the Berry flux by 2Wtt. 
Therefore, the Berry flux is only well defined up to mod 2n. Since the Berry flux is quantized to integer multiplied 
by 7r, only two classes of flux, 4>s = and tt, can be distinguished. All other values of $s can be connected to these 
two classes via redefinition of phase in equation (|S32I) . 

However, for a system with space inversion symmetry, if the contour was also chosen in such a way that it is 
invariant under space inversion (i.e. if k is a point on the contour, so is —k), then the Berry flux can be defined up 
to mod 4-7T. This can be achieved by requiring 

I^(f) = (S34) 

where / is the space-inversion operator. This is alway possible by choosing a proper (f> and redefine the Bloch 
waves as in equation (|S32I) . With this constraint, the redefinition of phase [equation (IS32[) ] can only be allowed for 
4>(k) = 2nir + 4>{—k) with n being an integer. Therefore, the winding number W [equation (|S33[) ] must be even, 
W = 2m with m being an integer. Thus, the Berry flux will now change by 4m7r under any phase redefinition. As 
a result, the Berry flux becomes well defined up to mod Air. Opposite to the general case discussed above, now the 
Berry flux 2tt and are topologically distinguished states and cannot be adiabatically deformed into each other. This 
is one example where a discrete symmetry of the system (e.g. the point group symmetry) changes the topological 
classification of a system. 

For the problem we study here, wg can always choose the eigenvector [d^m 

in equation (|S2|)] to be real. This 

enables the computation of the Berry flux via a simpler method. If one considers the phase uncertainty shown in 
equation f|S32[) as a gauge-like symmetry in the momentum space, this approach is in analogy to adopting a specific 
gauge which helps simplify the computation for physical quantities. We first require all the coefficients a„ jm to be 
real. However, there is still an arbitrary sign undetermined for a„, m at each k. To fix this sign, we choose a contour 
around the r or M point and require the wavefunction to satisfy the relation 

[^(r)Y*l(r)d 2 r = 1 + 0(\k - k'\), (S35) 

for any two momentums k and k' on the contour which are close to each other. Now, one still has the freedom to 
flip the sign for a n ^ rn for all the points on the contour at the same time, but up to this sign, the value of a n ,m is 
determined, which can be viewed as a gauge fixing. 

Due to the space inversion symmetry, the coefficients a n ^ m at k and — k must have the same amplitude but can 
have either the same or opposite signs: 

a n,m(k) — Q>n,m(~k), OT a n ^ m (k) = —CLn,m(~k). (S36) 



The Berry flux is zero for the first case but 27r for the second. 
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In the definition of the Berry flux, one uses the Bloch wavefunction in a specific band, say n. Therefore, the contour 
integral is well defined, when there is no band degeneracy on the contour. On the one hand, if the area enclosed by the 
contour has no band degeneracy point inside, one can continuously deform (shrink) the contour into a point, which 
by definition has zero Berry flux. During this adiabatic deformation of the contour, every quantity should change in 
a continuous way. On the other hand, the Berry flux is quantized and cannot be tuned adiabatically. Therefore, the 
Berry flux must be invariant during this process. These arguments lead to the following conclusion: for any contours 
enclosing no band degeneracy points for the band n, the Berry flux of this band must be zero. Now to the contrary, 
if the Berry flux is nonzero, this contour must not be able to shrink adiabatically to a point. The only way this is 
possible is when there is a band degeneracy point enclosed by this contour. Thus we prove that a nonzero Berry flux 
is the sufficient condition to have a protected band degeneracte point. 

We end the discussion about the topological nature of the band degeneracy by unifying the two topological indices 
that we have computed in this section. In fact, if one multiplies the winding number studied in Sec. IS-7 Al bv n, it 
coincides with the Berry flux examined in Sec. IS-7B1 



S-8. INSTABILITY INDUCED BY REPULSIVE INTERACTIONS 

The low-energy effective theory we constructed in Sec. IS-7 Al [equation (|S22[) ] can be mapped exactly to the general 
theory studied in Refs. [23|,[24j]. By mapping the results found in Ref. |23|| back into our model, the interaction effect 
can be solved in the weak-coupling limit. Here we summarize the results from this mapping and provide a simple 
mean-field picture for understanding this results. 

In the low-energy effective theory for fermions with two bands (but no spin), only one interaction term needs 
considering, 

H int = V^pl-Px^lfPyf. (S37) 

r 

Other (short-range) interaction terms involve derivatives and thus are irrelevant in the sense of renormalization group 
(RG). As shown in Ref. 23], at low temperature, this interaction is marginal at tree level under RG, while the one- loop 
RG calculation shows that this interaction is marginally relevant for V > (repulsive) and marginally irrelevant with 
V < 0. This implies that when the temperature is lowered for this system, for particles with attractive interactions, 
the noninteracting band structure theory remains accurate in the low energy limit. However, if the interaction is 
repulsive, it leads to an instability once the temperature is below certain critical value Tc . It worth emphasizing that 
the scaling behavior here is very similar to the BCS theory in Fermi liquids, except that the instability shows opposite 
behavior regarding the sign of interaction. In the latter case, the system is unstable for attractive interactions, but 
remains stable for repulsive interactions. 

The result of this instability is a state with spontaneously generated angular momentum. This can be seen if one 
notices that the interaction term in equation (|S37j) can be reformulated as 

^nt = -^]>>^ 2 ' (S38) 

f 

up to some unimportant constant where 

L% = -i{v\fVy.r - v\fVx,r)-, (S39) 

is the angular momentum in the z-direction at site r. The physical meaning of L z ~ can be checked by noticing 

L z \p x + ip y ) = +\p x + ipy), (S40) 
L z \p x - ip y ) = -\p x - ipy). (S41) 

Here we verify equation (IS38[) by expanding it using the fermion creation and annihilation operators: 

Hint =- -j ^{Lpf = J2^ p l:P p y^l:pPy^ ~ PlfPv&l.fP*,? ~ PifP^rPlfPy,? + v\^Px,rp\^Px,r)- (S42) 

r f 

Notice that the first and last term on the right hand side vanish because Jty - = p' jp' a = 0. The two remaining 
terms recover exactly the equation (|S37|) . with an extra constant term: VN/2 where N is the total particle number 
in the system. 
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From equation (IS38[) . it is transparent that repulsive interaction favors L z ^ 0, Since the repulsive interaction is 
marginally relevant, it shall dominate the low energy behavior of the system. Therefore, repulsive interaction shall 
result in the low-temperature instability towards a state with spontaneously generated angular momentum and (L z ) 
is the order parameter for this phase. The nonezero value of this order parameter indicates that the low-temperature 
ordered phase breaks spontaneously the time-reversal symmetry and also reduces the point group symmetry from 
D4 down to C2. Due to this symmetry breaking, the band degeneracy is no longer symmetry protected, and a gap 
opens at the band point. As a result, the system turns into an insulator with a finite energy gap. This insulator 
has a nontrivial topological structure, with the first Chern number being 1. The topological properties of this state 
is the same as the quantum Hall state with filling 1, but it is induced by interaction and spontaneous symmetry 
breaking, instead of a magnetic field. An insulating state of this kind of topological properties is often referred to as 
the quantum anomalous Hall state. [26] 

For larger V, although a Hartree-Fock treatment can still be preformed, the mean-field approximation becomes 
invalid as the energy scale of interaction becomes comparable with the band width. Furthermore, the low-energy 
effective theory, which we obtained by starting from a noninteracting band structure and treating interactions as 
perturbations, is not expected to describe the properties of a system with strong interactions. In this limit, the 
behavior of the system becomes sensitive to the microscopic details. 

We repeated the same mean-field calculation including the d-orbitals and all the conclusions above remains the 
same. As have been shown in Sec. IS-6l and lS-71 the energy of the d-orbitals are far from the Fermi level near the band 
degeneracy point and the degeneracy at T is between the two p-bands. Therefore, the interactions between d orbitals 
are not relevant for analyzing the instability of this level degeneracy point, and the same is true for interactions 
between p and d orbitals. The only relevant interaction term here is the repulsions between p orbitals, same as 
observed above in the two-band approximation. In the full three-band model, this interactions can be decoupled using 
the standard mean-field approximation as 



where Hq is the hopping part defined in Eq. (IS19[) and (LS) is the order parameter which are determined by minimizing 
the free energy. Same as the two-band model we computed above, this three-band model shows the same instability 
at infinitesimal V > 0. We compared other competing orders, such as nematic and density waves, and found that the 
topological insulating state is preferred. In the ordered phase with (Lp) ^ 0, we can use the mean- field Hamiltonian 
[Eq. (|S43[) ] to compute the energy spectrum. By solving the energy spectrum on a cylindrical geometry, we observed 
the topological edge modes as shown in Fig. 3 of the letter. 

S-9. DETECTING THE TOPOLOGICAL SEMIMETAL USING BRAGG SCATTERING 

Here, we briefly describe the experimental signatures of a semimetal state with respect to the application of Bragg 
scattering. When two laser beams with different frequencies and wavevectors are used, the particles in the system can 
absorb a photon from one beam and emit a photon into the other beam. In this process, the momentum gained by a 
particle confined in a 2D plane is Sq = feni — fc|| 2 , where feni and fc|| 2 are the in-plan components of the wavevectors 
of the two laser beams. In addition, the energy of the particle will change by Slo = u>i — L02 with lo\ and L02 being the 
frequencies of the beams. While lo can be tuned by changing the frequency of the two laser beams, Sq can be tuned 
independently via tilting the direction of the beams out of the 2D plane. Therefore, we can treat Sq and Slo as two 
independent control parameters. At a given value of Sq, as shown in Fig. IS5[ Bragg scattering is only allowed when 
Slo exceeds a threshold Slo*. The curves of Slo* as a function of Sq carry direct information on the band structure. 
For the topological semimetal, Slo* is a quadratic function of Sq and vanishes at Sq = 0. On the contrary, Slo* > for 
an insulator at any Sq, while Slo* = for a range of Sq in a metal. 
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FIG. S5: Experimental signatures for the topological semimetal and topological insulator in Bragg scattering. On the top row, 
the experimental setup of Bragg scattering is shown in (a) and the schematic scattering process is shown in (b) along with 
the formula for Sq and Sto. (c) and (d) demonstrate the scattering processes with small and large Suj respectively. The former 
processes is forbidden, due to the lack of a final state which can satisfy the energy and momentum conservation laws. The 
threshold for Suj above which the scatterings are allowed is shown in Figs, (e)-(g) as a function of the momentum change Sq, 
for a metal, topological semimetal and insulator accordingly. 



